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INTRODUCTION 



Private contractors can contribute very little to school district evaluations 
and worse yet, there is very little to which contributions could be made. Within 
the constraints imposed by the present organization of schools and the current 
methods of funding, the state of the art in school district research and evaluation 
is at a low level. Although the fiscal, theoretical and political pressures 
operatiiig upon and within districts all clearly point toward the need for a rigorous 
and sustained research and evaluation effort in the schools, the response to these 
challenges has been less than satisfactory. 

ISKternal contractors and school personnel are equally culpable for these 
failures. Increased participation of corporations, universities and individual 
consultants in the evaluation efforts of school districts will, despite and perhaps 
because of their externality, only prolong a condition that needs radical changing. 
However, this paper is not intended to be an obituary for the last and best attempt 
to bring reason to bear upon educational problems. Rather, it is a prescription for 
what might be done, once the problem has been explicated and easy solutions - such 
as the purported objectivity of the educational auditor or the presumed expertise 
of the private contractor ~ have been dismissed. 

PROBLEM DEFINITION 



The problems with current educational evaluation can be traced to two major 
sources: conceptual and administrative. The complexity and pervasiveness of these 
sources are such that no external evaluator or auditor can even begin to provide 
or suggest effective remedies. 

Conceptually, the applications of evaluation strategies and techniques lag 
far behind the models developed in the universities (Proves, 1969; Scrlven, 1967; 
Suchman, 1969). Many competent social scientists have tended to remain aloof 
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from applied educational evaluation, feeling that the lack of control over the 
variables and the political controversy surrounding accountability render the 
activity meaningless. Those researchers who have entered the field, regardless 
of the source of their institutional support, have brought more heart than wisdom 
to the enterprise. As a consequence, they have reinforced the practitioner's 
notion that there is one evaluation system, one evaluation ethos, and one payoff 
from evaluation. Educational evaluations continue to be constricted by the rigid 
laboratory model; insensitive to potential of - indeed, need for - ad hoc and 
intuitive approaches; and always pointed toward useful Information, as defined 
by the school business agent. 

In fact, evaluation is not a single concept so that one approach constitutes 
the universe. If one were to state the basic components of a comprehensive 

/ 

evaluation, there would be general agreement as to their nature. Statements of 
objectives, the documentation of processes, the specification of components, the , 

identification of outcomes, the description of the environment, the determination 

V 

of costs and provisions for unanticipated outcomes are the seven components most 
frequently mentioned. If these elements were consistently a part of e'^aluation 
in school districts, then the term evaluative research would be more appropriate. 

In applied settings, the experimental model cannot be static and the outcomes 
and their interpretations are rarely unambiguous. Evaluation then becomes an 
iterative process whereby successive approximations of the ultimate objectives are 
continually developed, refined, and questioned anew. In this approach, intuition 
is as informative as theory, targets of opportunity are as fruitful as pre- 
determined strategies, and the ambiance is as influential as the treatment. 

This approach to educational research is not anti-scientific nor is it non-rational. 
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It does, however, recognize the importance of situational variables in working 
toward improved programs that may not be generalized across schools or districts* 

If our evaluation priorities were realized so that evaluative research were 
the rule rather than the exception in school districts, external contractors 
would have very little to contribute to the schools for several reasons. Evaluative 
research takes time, years, if done properly, and it is senseless for districts 
not to hire the expertise rather than contract for it. Second, if the environment 
is as important as has been suggested, external evaluators would be Inadequately 
informed about these important variables. Finally, if evaluative research is 
as much a point of view as it is a system, then it ought to be instilled throughout 
the school system* Too often, evaluation is seen as somethirg that the planners 
*’do" and the administrators "use.” Evaluation should be a systemic perspective; 
its methods and results should become as important to the practitioners as to the 
administrators and planners* Unfortunat::ily, the use of external evaluators only 
reinforce the former, overly rigid conceptualization of the role of evaluation. 

Administratively, in those few districts where research and evaluation has 
been institutionalized, the divisions have been assigned a staff function. In 
this staff capacity, the personnel report either to an assistant superintendent 
or, in smaller districts, diiectly to the superintendent. Because districts are 
organized according to their operating units and special programs (Federal, E.S.E.A., 
Title I, etc.), there evolves an institutionalized separation between the planning- 
research-evaluation and the. teaching-learning functions. 

In part, this administrative separation can be attributed to the rigid 
adherence to the laboratory research concept discussed earlier in this paper. 

But even if the concept of evaluation in applied settings had been more flexible 



o 

ERIC 



4 



» 



and comprehensive, there are strong organizational pressures that would prevent 
its Impleinentation. The most important of these is the scarcity of qualified 
research and evaluation personnel. Schools of education have taken ever the 
training function, and their criteria and certification have by necessity been 
adopted by the school districts. The requisite courses in tests and measurement, 

"research design, statistics, etc. now must be taken from a university so that 
one may become an educational researcher. The universities have limited resources; 
consequently the supply of such people is limited. In turn, when scbcol districts 
hire their research personnel they are put in staff positions so that their talents, 

in theory, can be available to everyone. 

In reality what happens is that these people are then called upon to fight 
a continual series of brush-fires, most of which have very little to do with 
evaluation or research. They are called upon because they are quite capable, ^ 

fairly logical and articulate; and, after all, they are not really doing anything 
except planning an3Tway. The deterioration is rapid. As time is spent on other 

lay. 

tasks deemed critical by the superintendent, the research and evaluation staff is ^ 

increasingly separated from the operational staff. Finally, evaluation becomes 
something that is done once a year to satisfy some reporting requirement Imposed 
by some obscure source. 

Enter the external contractor who, for a price, will do the work to satisfy 
that report. And the work is tedious, expensive, and probably not satisfying to 
anyone. The original objectives of the program have to be stated and clarified 
again, for no one from the research and evaluation staff was available when 
principals and teachers planned the program. The treatments have long since been 
forgotten, because the key teachers have left and process monitoring was stopped. 

The data are in the janitor's supply closet, waiting to have the children s names 
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put on the instruments because they forgot to do that during the pre-testing. 

Finally, the post-testing is all set up for the contractor except for grade 
twelve which will be in Washington on their class trip. Through a variety of 
twists of the imagination that take him to, but not beyond, the boundary of 
perjury, the contractor produces the report and the requirement has been met. 

No one in the school district who cares about the quality of evaluation is 
satisfied, but there are more brush-fires to be put out so there is not enough 
time to worry about the problem. The need will arise next year and the contractor 
will probably be rehired because ”he can do the job.” 

As long as school districts are organized so that the research and evaluation 

« I 

people are treated as scarce resources that must be in a staff capacity, this 
depressing senario will be repeated. The planning, research, development and ^ 

diffusion functions will be neglected and probably dropped. Only the evaluation 
function will then remain and it will at best be performed mechanistically. 

I 

T 

The operating staff and the clients - teachers, students and parents - will look 
upon evaluation as something done by ^h®® b® u^* Of course, contractors may 
profit from this; but in the long run education loses. 

To counter this tendency, it is proposed that Indigenous research and evaluation 
resources be increased. However, along with this increase should come the 
administrative commitment of the staff to specific programs for their total life 
span. More resources that are used in the same old manner are not going to improve 
educational evaluations qualitatively. The time-consuming, institutional-maintenance 
demands of the system are great. The personnel of the research and evaluation 
groups will never satisfy these demands as long as they are in a purely staff 
position. Even if it means that some programs do not get evaluated, research 
and evaluation priorities must be established and specific individuals assigned 
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to the programs of interest. These individuals, it is suggested, should become 
participant-observers in their programs, even to the point of teaching classes. 
Furthermore, while they learn about the realities of the schools from their peers 
and from direct experience, they should look upon their role as one that is 
overtly pedagogical. That is, the perspective and the few techniques that are 
connoted by the term evaluative research are to be taught to the operating staff 
as the program progresses. Instilling more of the line responsibilities into 
research and evaluation groups is one way to initiate a process in which the 
planning, research, development and evaluation functions are perseverated throughout 
the system. Obviously, external evaluation contractors cannot contribute to this 
necessary process. 

NEW DIRECTIONS 

If research were to become an integral part of school district evaluation, 
the opportunities for extending the state of the art would be limitless. There 
are two area«, however, in which development would have a particularly high 
priority because of their importance to applied evaluation. The first is in 
developing a methodology for clarifying and systematizing formative evaluations. 

The second is in improving the attitude survey capabilities of school districts. 

It is not presumed that research activity in two areas will solve the problems 
of education. However, it is suggested that work in these areas will Increase 
our ability to ask questions that are at least pertinent to the problems. 

FORMATIVE EVALUATION IMPROVEMENT 

One of the principal problems in conducting formative evaluations of complex 
problems is the inherent difficulty of organizing the wealth of material in such 
a way as to throw light directly upon the issues of interest. Thus the first 
step in formative evaluation Improvement would be the examination of the agreed- 
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upon evaluation objectives and the explication of those which relate to the 
formative evaluation. 

This explication will involve reducing the general objectives to specific 
categories into which information will be classified. This categorization step 
is necessary to avoid the frequently encountered program evaluation situation in 
which a large amount of data and great piles of documents are accumulated with 
no clear plan for the use of the data. It then becomes necessary to reduce the 
Information on an a posteriori basis to some sensible format. In that process, 
it is usually found that the data are not comparable among projects, some of the 
data are redundant, or that important evaluation objectives have been ignored- 
and the data were not collected. It is believed that a great deal of time can 
be saved and comprehensive evaluation coverage can be insured by structuring in ^ 

advance the content categories in which information will be collected and maintained 
for the formative evaluation. 

Even with a thorough planning function built into school evaluations, one of 
the primary difficulties in conducting formative evaluation is the formating of 
information in such a way as to show systematically the interaction effects of the 
various program components. It is proposed that this problem would be overcome by 
the development and implementation of the §22— -S Diagram approach to overall 

program concepts for each of the projects in the program. This technique allows 
one to describe the various components and their relationships in the overall program 
in such a way that every one concerned can examine the description, correct it 
where there is an error, change it as programmatic changes are introduced, and 
understand the relationship among the various activities within each component. 

Figure 1 illustrates how such a system block diagram can be used to describe 
the components of a program. This example is taken from a previous project in 
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which Planar personnel had the responsibility for describing various components 
in an employee recruiting and training program. The component illustrated has 
to Co with the training function. 

Supporting the diagrammatic description of each project would be a series 
of back-up sheets giving a narrative description of the specific details underlying 
each activity shown in the rectangular boxes on the diagram. In the example 
presented, the back-up narratives described the processes by which the instructor 
and the field counselor provide orientation to the new employees, detailing the 
materials used; all course outlines that might be used; the schedule followed; 
qualifications of the instructor and the field counselor; and a statement of the 
specific goals and objectives of the orientation section. In school districts 
this might be analogous to the orientation of the parents to a school-wide program. 
It can be seen that the availability of the diagrammatic descriptions together with 
the backup material can provide a uniform way of examing the various programs 
so as to pinpoint communalities and differences. If it were compared with a 
**model" project, for example, it would facilitate the comparison of objectives 
and components and the identification of disparities between the actual projects 
and the "model" as well as among projects. 

Both In the diagramming and in the descriptive material supporting the 
diagram there would be special emphasis on the mechanisms provided for feeding 
back Information for program improvement and quality control. These feedback 
loops can be shown in the diagrams and specific techniques for implementation of 
feedback mechanisms would be detailed in the written descriptions. 

Additional documentation which would support the formative evaluation would 
Include material bearing on environmental factors in the nation which tended 
to foster successful development of the programs or to hinder tneir orderly 
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development. These would irnlude minutes of meetings, public correspondence, 
newspaper or other media coverage, and the information culled from on-site 
interviews of the staff and community. In addition, they would include the 
direct experience observations of the research staff member assigned to the 
evaluation as a participant-observer. 

The availability of the program descriptions and the iterative comparison 
of such program descriptions against the program objectives will provide a con- 
tinuing opportunity to review the congruence of program elements and to modify 
processes and/or components as necessary. Formative evaluation is in its infancy 
and is far from being a science in any sense of the word. Unless and until the 
data are presented in a clear and detailed manner, all of the techniques in the 
evaluator's kit x^ill not begin to address the salient questions. System Block 

; / 

Diagrams do clarify the data by displaying the key program components in their ; , 

natural sequence. The back-up narrative material provides the essential contextual 
Information. Taken together, these data are useful to the evaluator simply 

V 

because they are organized. 

ATTITUDE SURVEY IMPROVEMENT 

Because educational issues are increasingly brought to the political forum 
and because taxpayers* revolts are becoming more frequent, school people are 
ever more dependent upon the timely and accurate assessment of attitudes toward 
education. In large school districts, it is suggested that the only way educators 
can be attuned to public opinion while remaining aloof from becoming a captive of 
a vocal minority is to develop systematically an Internal survey capability. 

Such a capability, developed over time by the research and evaluation staff, 
would at the same time have obvious programmatic benefits. One of the reasons 
why schools are judged by standardized test scores is that they are often the only 
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credlble data to which the public has access. Because of a lack of emphasis 
upon survey research, schools have never been able to use, in either public 
relations or program evaluation, attitudinal data that hav® great promise. 

Both as outcome and intervening variables, attitudinal information would be 
restored to its important position, if the instrumentation, collection and analysis 
functions were treated systematically. 

The most productive role systems oriented social scientists can play in 
supporting a school district’s planning and evaluation activities in this realm 
is through the development, refinement, and field testing of methods and techniques 
for the conduct of analyses in support of decision making activities. These 
methods and techniques could, of course, cut across many fields ranging from 
highly sophisticated computer modeling and simulations to highly specific studies 
of the relative impact of various incentives for the enhancement of academic 
performance. The quality of decisions is, of course, directly related to the 
quality of information available to decision makers. What is here proposed is 
that a program of research and development be undertaken, the objective of which 
is the ultimate production of survey Instruments and analytical techniques which 
have the following characteristics: 

1. Have been shown to be highly reliable, l.e., to produce essentially 
Identical results when administered successively to the same sample 
or to comparable samples. 

2. To be sensitive Indicators of the direction and strength of a wide 
variety of opinions, attitudes and feelings of school district 

personnel* 

3. Are valid, l.e., can be shown empirically to measure variables which 
relate significantly to behaviors important to the school district. 

Surveys designed to tap such variables are, of course, in widespread use 

i 

throughout school districts. Such surveys have been, and will continue to be, 
useful sources of relevant background information. It is suggested, however, 

that the typical survey of this type is not designed so as to optimize the 
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probability of producing the most useful information. This is partly a function 
of the fact that most of these studies are conducted on an ad hoc basis, usually 
by personnel who have limited access to or experiences with the most sophisticated 
techniques. In addition, it is seldom possible in one-time studies of this type 
to collect validation data on a systematic basis and to accumulate a body of such 
validation data on the same scales over extended periods of time. All of these 
factors mitigate against significant improvement in the state of the art with 
respect to surveys designed to produce useful information for decision makers* 

The primary methodologies involved would be the conduct of both cross- 
sectional and longitudinal surveys in a variety of content areas and the analysis 
of responses obtained for relationships with behavioral criteria. In the course 
of these surveys all promising techniques of data collection through interviews 
and observations, all promising procedures for reducing data to numerical values 
through scaling, all likely statistical procedures for data processing, and all 
likley techniques for synthesizing and reporting of results will be systematically 
evaluated. Thus, over a period of time the optimum configuration of data collection, 
processing, and analysis will be developed and the limits of reliability and 
validity will be demonstrated. 






^^hat is proposed is a program of research and development. This program 
would include the conduct of several studies each, of which would be designed ■ 

to capitalize upon the findings of previous studies and to advance the state of | 

I 

the art of surveying within the educational setting. Since the specific studies 
to be undertaken Wcild be predicated on the needs of the district and on previous 
findingSf it is impossible to present a definitive program. It is however, clear 
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that the typical study would proceed generally as displayed in Figure 2. The 
major steps would be as follows: 

The first step would be to identify the problem area in which important 
policy decisions can be expected to be required in the near future. 

Second would be the identification of those district sub-populations whose 
opinions, attitudes, and feelings are important inputs to the decisions to be 
made. These might be categories of administrators or teachers, age groups, 
groups with differing lengths of employment, men vs. women, etc. Once these 
populations of concern are identified and described, appropriate sampling 
techniques would be employed to derive the samples to be utilized. In every 
case the samples should be large enough to provide useful data for decision 
makers and to provide the capability for analyses for the refinement of method 
and technique. 

Third, the extensive literature on survey research would be reviewed to 
Identify all possible techniques for data collection which show promise in the 
type of study to be conducted. From a methodological point of view it is important 
to over-sample rather than under-sample techniques and methods to be utilized. 

Thus, there would be a minimum likelihood of overlooking techniques which may 
contribute significantly to the precision of results. 

Fourth, field surveys would be undertaken with samples of the populations 
selected using the chosen techniques. It will normally not be necessary to 
administer all subsets of all techniques to all subjects in the samples. Where 
relatively large samples are available and necessary subsets of the item pools 
properly counterbalanced to allow later statistical treatment, the surveys can be 
administered in such a way that no one subject is unduly burdened. Powerful 
computerized statistical techniques are now available for dealing with data cells 
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wich different numbers of cases in the analysis phases. While the exact details 
of each field survey will vary with the content, the respondents Involved, and 
the specific purposes of the survey, it seems clear that survey facilities and 
procedures should be comparable to those which would be utilized by the school 
district under normal circumstances rather than those employing highly sophisticated 
and unusually well— trained survey personnel. This is necessary to insure that 
findings with respect to methodology and technique will be generalizable to normal 
survey operations to be conducted after this developmental effort is completed. 
Occasionally, of course, it will be necessary for project personnel to conduct 
small sample studies with highly specialized survey personnel in order to resolve 
especially complex methodological problems. 

Next, survey results will be subjected to a variety of analytical and data / 

processing techniques. These will be considerably more detailed and varied than 
would be the case in a normal survey situation because one of the goals of the 
project is to evaluate empirically the relative merits of various proposed V 

alternatives for such data treatment. 

Finally, survey results will be synthesized and presented in a variety of 
report formats. The goal here will be to evaluate the effectiveness of various 
alternative techniques for displaying survey results to the public or to decision 
makers. It is likley that the quality of decisions varies not only as a function 
of the accuracy of data available but to some extent as a function of the forniat 
in which the Information is presented. Certain data may be most understandable 
in numerical form - others in pictorial displays and still others in combinations 
supported by prose summaries. Again, one of the goals of the project will be 
systematically to vary data display techniques and to obtain empirical (at least 
judgmental) evidence as to the relative value of the alternatives. 
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The methodological implications of each particular survey would then be 
studied in detail and be made a part of the overall body of information the 
program is accumulating. This will involve examining the outcomes (particularly 
those relating to technique) in light of findings from similar surveys which are 
part of the overall program. It is naturally important that specific surveys 
involve subject matter areas of interest to the district at the time. Thus, the 
program will have the double benefit of working toward the ultimate optimization 
of survey technology and of providing, on a continuing basis, live data to those 
responsible for making decisions while the program is underway. 

CONCLUSION 

Evaluative research as a world-view is too important to be left to the 
evaluators. The narrow focus upon externality as the only evaluative perspective, 
the emphasis upon formal analytical techniques rather than logical design, and 
the tendency to let evaluation for merit subsume the planning and research functions 
all perpetuate an unhealthy climate in which some evaluation contractors prosper. 

It is proposed that new infusions of money along with a broader definition of 
evaluation and an administrative restructuring of evaluation activities hold the 
promise of changing the system. Hox<rever, it takes time to make these changes 
and they are best done internally. External contractors can only siphon off 
needed money while using whatever skill they have in perpetuating the inadequate, 
fragmented efforts of the past. 

Two new directions for strengthening the evaluation and research efforts of 
school districts have been discussed. Systematic approaches to Improving formative 
evaluation and attitude survey techniques will allow school districts to ask the 
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right questions in educational evaluation. By reordering their priorities and 
their administrative procedures, school districts would be able to put the 
research back into research and evaluation. Certainly no external group or 
individual is going to do this for the schools . 
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